Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SRTM Version 3 Support #39

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Conversation

nawagers
Copy link
Contributor

New support for SRTM Version 3, Global 1 arc.
The new SRTM v3 is completely void filled and offers 1 arc second resolution tiles over the entire world (between ~56S and ~60N). This addresses issue #38. Users need to obtain Earthdata credentials from NASA Earthdata New User to make use of the new functionality.

Changes

  • get_data() and main.py: The FileHandler class was moved to data.py due to a previously unknown bug and circular dependencies. get_data() was updated to add version and fallback parameters as well as deprecating srtm1 and srtm3 usage.

  • EarthData: A new class was added from NASA's programming guide. It's a subclass of requests.Session that overrides the rebuild_Auth function to keep authentication headers when interacting with their server.

  • Tile Version: A version system was added for the tiles to keep track of which data version (1, 2.1, and 3) and which sampling (1 arcsecond, 3 arcsecond averaged, and 3 arcsecond sampled). The versioning applies to the in memory tile store and the local disk cache. Requests for elevation now accept a version. It's important to know that this will force the old cache to rebuild.

  • Memory Handling: The tiles are all saved in the dictionary named tiles with keys of the form 'N00E000v2.1a'. This is a class attribute and so is shared between all instances. This was also true before, but __init__ would reset the dict, so every new object would flush the memory. The index, called tile_index, is also a class attribute dictionary. If the instance sees that the dict is empty during __init__, it will load it back from srtm.json. The keys to the index are of form 'N00E000' and holds a list of version information. The values in each list correspond to the tile lookup for each version and sampling.

  • Fallback: A bool, GeoElevationData.fallback, turns on and off the fallback behavior. When True, if the requested version can't be loaded, the next version is attempted. This can happen if a server is down or the user supplied bad credentials for version 3 data. The version resolution is laid out in fallback_version(). Users can substitute their own order by subclassing GeoElevationData or saving a new function to the instance's function. An example is in the docstring. The default version is v2.1a which drops to v2.3a and then fails. This is the same behavior as before.

  • load and fetch: The internal function calls were reorganized to a cleaner and private interface with the functions _get_tilename, _load_tile, _fetch, and _build_url. The other functions that handled similar actions before were deprecated and in some cases, updated to make use of the new functionality.

  • Travis CI: Dropped support for 2.6. Builds on 2.6 are fragile and restrict language features. Dependency 'Requests' dropped support for 2.6. Added testing for 3.6 and now runs coverage, coveralls, pyflakes, and pycodestyle. The new EarthData credentials are stored in environment variables and loaded by test.py

  • Test Files: Added a test file folder and three ver2 3 arcsecond tiles. test.py now has a function that replaces fetch() to simulate downloading and reduce network dependency for tests. The new tests written make use of this (except the test for fetch() itself).

  • Unit tests: The test for batchmode was updated for some small syntax changes and changed some of the assertions to give more helpful failure messages. A comprehensive unittest was added for each of the new functions in GeoElevationData.

  • srtm.json: A new dict structure was added as srtm.json to replace list.json. The new structure includes information on versions 1, 2.1, and 3 for 1 arc, 3 arc averaged and 3 arc sampled tiles.

  • Backwards Compatibility: It is expected that the new package code should be completely backwards compatible except for users on Python 2.6 and scripts that directly modified instance and class attributes (such as srtm1_files).

  • Dev Tools: A script to build the json (or pickle) and the tile lookup for each version are included in the dev_tools directory.

nawagers and others added 9 commits December 17, 2017 00:39
deleting *.hgt and *.zip
Using rename instead of replace (could cause errors on windows in the test script if a test previously failed to cleanup). Also changed super() args needed for older versions
GeoElevationData depends on FileHandler, call in module main depend on module data. This creates a circular dependency. Moved FileHandler to the module data to fix. Added unittest for get_elevation. Adjusted unittests for the new FileHandler location.
Set UN and PW in travis, added 3.6, run coverage, coveralls, pyflakes and pycodestyle. Small format changes is data.py. Docstring changes in test.py and data.py.
Python 2.6 support has been dropped by Requests (dependency) and restricts language features. Fixed typo in test.py
Added dev_tools folder for generating srtm.json. Updated or added docstrings and minor code format changes. Cleaned up main.get_data() to remove unneeded code. Made several new functions private (not intended for public API)
@tkrajina
Copy link
Owner

Conflicts in master resolved in https://github.com/tkrajina/srtm.py/tree/nwagers-EarthDataLogin

@SteveAlexander
Copy link

just a note that I'm using this branch, and haven't experienced problems with it yet

@berndhahnebach
Copy link

Is there any special reason why this is not in master?

So I need to get a login to use this data set?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants